25 research outputs found
Low-Rank Projections of GCNs Laplacian
In this work, we study the behavior of standard models for community
detection under spectral manipulations. Through various ablation experiments,
we evaluate the impact of bandpass filtering on the performance of a GCN: we
empirically show that most of the necessary and used information for nodes
classification is contained in the low-frequency domain, and thus contrary to
images, high frequencies are less crucial to community detection. In
particular, it is sometimes possible to obtain accuracies at a state-of-the-art
level with simple classifiers that rely only on a few low frequencies
Population-Based Reinforcement Learning for Combinatorial Optimization
Applying reinforcement learning (RL) to combinatorial optimization problems
is attractive as it removes the need for expert knowledge or pre-solved
instances. However, it is unrealistic to expect an agent to solve these (often
NP-)hard problems in a single shot at inference due to their inherent
complexity. Thus, leading approaches often implement additional search
strategies, from stochastic sampling and beam-search to explicit fine-tuning.
In this paper, we argue for the benefits of learning a population of
complementary policies, which can be simultaneously rolled out at inference. To
this end, we introduce Poppy, a simple theoretically grounded training
procedure for populations. Instead of relying on a predefined or hand-crafted
notion of diversity, Poppy induces an unsupervised specialization targeted
solely at maximizing the performance of the population. We show that Poppy
produces a set of complementary policies, and obtains state-of-the-art RL
results on three popular NP-hard problems: the traveling salesman (TSP), the
capacitated vehicle routing (CVRP), and 0-1 knapsack (KP) problems. On TSP
specifically, Poppy outperforms the previous state-of-the-art, dividing the
optimality gap by 5 while reducing the inference time by more than an order of
magnitude
There Is No Turning Back: A Self-Supervised Approach for Reversibility-Aware Reinforcement Learning
We propose to learn to distinguish reversible from irreversible actions for
better informed decision-making in Reinforcement Learning (RL). From
theoretical considerations, we show that approximate reversibility can be
learned through a simple surrogate task: ranking randomly sampled trajectory
events in chronological order. Intuitively, pairs of events that are always
observed in the same order are likely to be separated by an irreversible
sequence of actions. Conveniently, learning the temporal order of events can be
done in a fully self-supervised way, which we use to estimate the reversibility
of actions from experience, without any priors. We propose two different
strategies that incorporate reversibility in RL agents, one strategy for
exploration (RAE) and one strategy for control (RAC). We demonstrate the
potential of reversibility-aware agents in several environments, including the
challenging Sokoban game. In synthetic tasks, we show that we can learn control
policies that never fail and reduce to zero the side-effects of interactions,
even without access to the reward function
Geometric Deep Reinforcement Learning for Dynamic DAG Scheduling
International audienceIn practice, it is quite common to face combinatorial optimization problems which contain uncertainty along with non-determinism and dynamicity. These three properties call for appropriate algorithms; reinforcement learning (RL) is dealing with them in a very natural way. Today, despite some efforts, most real-life combinatorial optimization problems remain out of the reach of reinforcement learning algorithms. In this paper, we propose a reinforcement learning approach to solve a realistic scheduling problem, and apply it to an algorithm commonly executed in the high performance computing community, the Cholesky factorization. On the contrary to static scheduling, where tasks are assigned to processors in a predetermined ordering before the beginning of the parallel execution, our method is dynamic: task allocations and their execution ordering are decided at runtime, based on the system state and unexpected events, which allows much more flexibility. To do so, our algorithm uses graph neural networks in combination with an actor-critic algorithm (A2C) to build an adaptive representation of the problem on the fly. We show that this approach is competitive with state-of-the-art heuristics used in high-performance computing runtime systems. Moreover, our algorithm does not require an explicit model of the environment, but we demonstrate that extra knowledge can easily be incorporated and improves performance. We also exhibit key properties provided by this RL approach, and study its transfer abilities to other instances
Better state exploration using action sequence equivalence
International audienceIncorporating prior knowledge in reinforcement learning algorithms is mainly an open question. Even when insights about the environment dynamics are available, reinforcement learning is traditionally used in a tabula rasa setting and must explore and learn everything from scratch. In this paper, we consider the problem of exploiting priors about action sequence equivalence: that is, when different sequences of actions produce the same effect. We propose a new local exploration strategy calibrated to minimize collisions and maximize new state visitations. We show that this strategy can be computed at little cost, by solving a convex optimization problem. By replacing the usual ϵ-greedy strategy in a DQN, we demonstrate its potential in several environments with various dynamic structures
READYS: A Reinforcement Learning Based Strategy for Heterogeneous Dynamic Scheduling
International audienceIn this paper, we propose READYS, a reinforcement learning algorithm for the dynamic scheduling of computations modeled as a Directed Acyclic Graph (DAGs). Our goal is to develop a scheduling algorithm in which allocation and scheduling decisions are made at runtime, based on the state of the system, as performed in runtime systems such as StarPU or ParSEC. Reinforcement Learning is a natural candidate to achieve this task, since its general principle is to build step by step a strategy that, given the state of the system (the state of the resources and a view of the ready tasks and their successors in our case), makes a decision to optimize a global criterion. Moreover, the use of Reinforcement Learning is natural in a context where the duration of tasks (and communications) is stochastic. We propose READYS that combines Graph Convolutional Networks (GCN) with an Actor-Critic Algorithm (A2C): it builds an adaptive representation of the scheduling problem on the fly and learns a scheduling strategy, aiming at minimizing the makespan. A crucial point is that READYS builds a general scheduling strategy which is neither limited to only one specific application or task graph nor one particular problem size, and that can be used to schedule any DAG. We focus on different types of task graphs originating from linear algebra factorization kernels (CHOLESKY, LU, QR) and we consider heterogeneous platforms made of a few CPUs and GPUs. We first propose to analyze the performance of READYS when learning is performed on a given (platform, kernel, problem size) combination. Using simulations, we show that the scheduling agent obtains performances very similar or even superior to algorithms from the literature, and that it is especially powerful when the scheduling environment contains a lot of uncertainty. We additionally demonstrate that our agent exhibits very promising generalization capabilities. To the best of our knowledge, this is the first paper which shows that reinforcement learning can really be used for dynamic DAG scheduling on heterogeneous resources
MetaREVEAL: RL-based Meta-learning from Learning Curves
International audienceThis paper addresses a cornerstone of Automated Machine Learning: the problem of rapidly uncovering which machine learning algorithm performs best on a new dataset. Our approach leverages performances of such algorithms on datasets to which they have been previously exposed, i.e., implementing a form of meta-learning. More specifically, the problem is cast as a REVEAL Reinforcement Learning (RL) game: the meta-learning problem is wrapped into a RL environment in which an agent can start, pause, or resume training various machine learning algorithms to progressively "reveal" their learning curves. The learned policy is then applied to quickly uncover the best algorithm on a new dataset. While other similar approaches, such as Freeze-Thaw, were proposed in the past, using Bayesian optimization, our methodology is, to the best of our knowledge, the first that trains a RL agent to do this task on previous datasets. Using real and artificial data, we show that our new RL-based meta-learning paradigm outperforms Free-Thaw and other baseline methods, with respect to the Area under the Learning curve metric, a form of evaluation of Anytime learning (i.e., the capability of interrupting the algorithm at any time while obtaining good performance)
Interferometric Graph Transform for Community Labeling
We present a new approach for learning unsupervised node representations in community graphs. We significantly extend the Interferometric Graph Transform (IGT) to community labeling: this non-linear operator iteratively extracts features that take advantage of the graph topology through demodulation operations. An unsupervised feature extraction step cascades modulus non-linearity with linear operators that aim at building relevant invariants for community labeling. Via a simplified model, we show that the IGT concentrates around the E-IGT: those two representations are related through some ergodicity properties. Experiments on community labeling tasks show that this unsupervised representation achieves performances at the level of the state of the art on the standard and challenging datasets Cora, Citeseer, Pubmed and WikiCS
READYS: A Reinforcement Learning Based Strategy for Heterogeneous Dynamic Scheduling
International audienceIn this paper, we propose READYS, a reinforcement learning algorithm for the dynamic scheduling of computations modeled as a Directed Acyclic Graph (DAGs). Our goal is to develop a scheduling algorithm in which allocation and scheduling decisions are made at runtime, based on the state of the system, as performed in runtime systems such as StarPU or ParSEC. Reinforcement Learning is a natural candidate to achieve this task, since its general principle is to build step by step a strategy that, given the state of the system (the state of the resources and a view of the ready tasks and their successors in our case), makes a decision to optimize a global criterion. Moreover, the use of Reinforcement Learning is natural in a context where the duration of tasks (and communications) is stochastic. We propose READYS that combines Graph Convolutional Networks (GCN) with an Actor-Critic Algorithm (A2C): it builds an adaptive representation of the scheduling problem on the fly and learns a scheduling strategy, aiming at minimizing the makespan. A crucial point is that READYS builds a general scheduling strategy which is neither limited to only one specific application or task graph nor one particular problem size, and that can be used to schedule any DAG. We focus on different types of task graphs originating from linear algebra factorization kernels (CHOLESKY, LU, QR) and we consider heterogeneous platforms made of a few CPUs and GPUs. We first propose to analyze the performance of READYS when learning is performed on a given (platform, kernel, problem size) combination. Using simulations, we show that the scheduling agent obtains performances very similar or even superior to algorithms from the literature, and that it is especially powerful when the scheduling environment contains a lot of uncertainty. We additionally demonstrate that our agent exhibits very promising generalization capabilities. To the best of our knowledge, this is the first paper which shows that reinforcement learning can really be used for dynamic DAG scheduling on heterogeneous resources